[python] Add paired ACT storage benchmark by YannByron · Pull Request #9466 · apache/paimon

YannByron · 2026-08-29T14:44:52Z

Summary

Add a reproducible paired ACT benchmark for equivalent RoboMIND data stored in HDF5 and Paimon.

This PR now contains only the benchmark. It builds on the snapshot-pinned ContiguousWindowDataset merged in #9580.

Changes

Add prepare, run, and compare commands for independent HDF5 and Paimon runs using one resolved experiment.
Verify source identity, logical window order, tensors, and loss traces before comparing performance metrics.
Use frame_index consistently as the ACT sample position across both storage backends.
Package the default experiment and optional ACT benchmark dependencies.
Document the workflow, reported metrics, and measurement boundaries.

Testing

Rebased onto the latest master (d95ebff28).
Full Python suite: 4701 passed, 78 skipped on Python 3.13.
ACT and RoboMIND regression suite: 48 passed, 1 skipped.
Flake8, license headers, compileall, and git diff --check.
RED → GREEN: decimal-valued HDF5 actions reproduced the statistics-domain mismatch before the fix and pass after canonical float32 conversion.

Notes

Ingestion and canonical-action backfill are outside timed benchmark runs.
The benchmark does not control the OS page cache.
GPU, multi-worker loading, distributed training, recovery, and policy quality are outside this local CPU benchmark.

JingsongLi · 2026-09-03T08:49:25Z

Thanks for putting this together. This PR currently combines two independently valuable changes with different review and risk profiles: the generic ContiguousWindowDataset public API, and the RoboMIND ACT benchmark harness and workflow.

Given the 4.4k-line scope, could we split this into two PRs?

Contiguous-window Dataset: the public API, row-ID and snapshot semantics, BLOB handling, batching behavior, tests, and API documentation.
ACT benchmark: the prepare/run/compare workflow, backend adapters, packaging, benchmark documentation, and ideally a real paired Vortex result artifact with variance and physical-layout identity.

This would make the public API easier to review on its own, while allowing the benchmark methodology and reproducibility contract to be evaluated independently. It would also keep follow-up fixes in either area from blocking the other.

JingsongLi

Three actionable issues found in the updated benchmark-only scope.

Rebuild the benchmark-only change on top of the merged contiguous-window dataset. Compare equivalent HDF5 and Paimon ACT inputs and use frame_index consistently in the public sample contract. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 181/2395 AI-Contributed/UT: 687/895

Align RoboMIND fields and formats with the public data contract, compare action statistics in the canonical numeric domain, and keep validation reads outside benchmark timing. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 AI-Contributed/Feature: 71/71 AI-Contributed/UT: 57/57

JingsongLi · 2026-09-09T11:52:09Z

+1

YannByron force-pushed the m0/pr3-act-benchmark branch from 622ceae to 5d8890e Compare August 29, 2026 14:53

YannByron marked this pull request as draft August 29, 2026 14:54

YannByron force-pushed the m0/pr3-act-benchmark branch 2 times, most recently from fb77b0e to 7b2b5a4 Compare September 1, 2026 03:24

YannByron marked this pull request as ready for review September 1, 2026 03:51

YannByron marked this pull request as draft September 1, 2026 03:52

YannByron marked this pull request as ready for review September 2, 2026 01:14